Fundamentals of Missing Data in Evaluation

Presentation to MSU Department of Psychology, Program Evaluation Occasional Speaker Series, East Lansing, MI

Steven J. Pierce

pierces1@msu.edu

Center for Statistical Training and Consulting

2024-12-05

Outline

What is missing data?
Why do we end up with missing data?
Why do we care about missing data?
Describing missing data
- Mechanisms
- Amount
- Patterns
What should we do about missing data?
- Prevention
- Diagnosis
- Treatment
Advice

What is missing data?

Missing data (MD) are measurements you intended to collect but did not get.

Having MD is common in research & evaluation studies.
If you do much evaluation work, you will run into MD.

Why do we end up with missing data?

Data collection doesn’t always go according to plan…

Human Factors	Other Factors
Participant behavior	Equipment failures
Evaluator errors	Records/Databases
Partner behavior	Unusual Events

Like many things in life, data collection doesn’t always go according to plan. Human factors often contribute to the occurrence of missing data. For example, a survey participant may skip a question without recording any response, offer an illegible or irrelevant answer, refuse to answer it, or simply say “I don’t know”. They may skip whole instruments or entire data collection visits. But evaluators occasionally make errors that lead to missing data. Imagine forgetting to take a data collection form with you, or programming the skip pattern for an online survey incorrectly. Sometimes, our evaluation partners’ decisions or behavior can prevent obtaining some data we wanted. Perhaps they object to sharing something with you, or were not diligent about logging some events of interest.

Other things can go wrong too. Perhaps a physical instrument breaks, or its battery dies, or your online survey app crashes. If you’re using an organization’s records or databases, those may be incomplete, difficult to search for relevant data, coded in unhelpful ways, or could have been lost or damaged or even destroyed according to some retention policy. In terms of events, consider the recent hurricanes in Florida that disrupted data collection for one of my projects.

Why should we care about missing data?

Ethics: Guiding Principles for Evaluators

Handling missing data well enacts our guiding principles^[1]:

AEA logo.

Systematic inquiry
Competence
Integrity

Perhaps the broadest answer to why we should care about missing data is that this is an ethics issue. Thoughtful handling of missing data is a good way to enact several of AEA’s guiding principles for evaluators.

> Systematic inquiry demands that we examine the quality of the available data and use it appropriately. We have an obligation to use sound methodology. There are scientific consequences of missing data but there are also legitimate, evidence-based methods for minimizing its impact on our analyses, conclusions, and products.
> Competence demands that we invest in skills necessary to understand and handle missing data properly because doing it poorly may compromise our evaluations. Our funders, clients, and stakeholders–including the program staff and the people and communities they serve and operate within–need and deserve competent evaluation services.
> Integrity demands that we be transparent about the amount of missing data, how we handled it, and how it may affect our evaluation results.

Consequences

Bias

In appropriate handling of missing data can cause analyses to yield biased results.

Generalizability

Misalignment of analyzed sample and intended population

Power

Most statistical software defaults to listwise deletion of cases that have any missing values on the variables involved in an analysis. That reduces statistical

^[2,3]

Types of Missingness

Item-level
Construct
Person-period
Planned vs unplanned

Mechanisms of Missingness

Missing completely at random (MCAR)
Missing at random (MAR)
Missing not at random (MNAR)

MCAR

MCAR is when neither observed nor unobserved variables predict which data is missing

MAR

MNAR

Diagnosis

Describing the Amount of MD

^[3]

Numbers & percentages of complete & incomplete cases
Number and percentage of missing values for each variable
Nature and frequency of missing data patterns

Predictors of Attrition & Missingness in Longitudinal Studies

Study arm: Compare retention rates
Study site (in multisite studies)
Baseline/pretest values of outcome variables may predict who drops out or has missing values
Other covariates (demographics, site, )

Treatment

Prevention

An ounce of prevention is better than a pound of cure

^[4],^[5]

Advice

Collaborate with a statistician!

Practical Options

Item-level missingness in scale scores^[6,7]

References

1. American Evaluation Association. (2018). Guiding principles for evaluators [Web Page]. Author. https://www.eval.org/About/Guiding-Principles

2. Fernández-García, M. P., Vallejo-Seco, G., Livácic-Rojas, P., & Tuero-Herrero, E. (2018). The (ir)responsibility of (under)estimating missing data. Frontiers in Psychology, 9(556). https://doi.org/10.3389/fpsyg.2018.00556

3. McKnight, P. E., McKnight, K. M., Sidani, S., & Figueredo, A. J. (2007). Missing data: A gentle introduction. Guilford Press.

4. Leeuw, E. D. de. (2001). Reducing missing data in surveys: An overivew of methods. Quality & Quantity, 35(2), 147–160. https://doi.org/10.1023/A:1010395805406

5. Wisniewski, S. R., Leon, A. C., Otto, M. W., & Trivedi, M. H. (2006). Prevention of missing data in clinical research studies. Biological Psychiatry, 59, 997–1000. https://doi.org/10.1016/j.biopsych.2006.01.017

6. Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549–576. https://doi.org/10.1146/annurev.psych.58.110405.085530

7. Newman, D. A. (2014). Missing data: Five practical guidelines. Organizational Research Methods, 17(4), 372–411. https://doi.org/10.1177/1094428114548590